Papers with model complexity
FT-MDT: Extracting Decision Trees from Medical Texts via a Novel Low-rank Adaptation Method (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods for extracting medical decision trees rely on manual annotation . PI-LoRA is a low-rank adaptation method for extract medical decision tree from clinical guidelines and textbooks . |
| Approach: | They propose a low-rank adaptation method for automatically extracting medical decision trees from clinical guidelines and textbooks. |
| Outcome: | The proposed method outperforms existing methods for the Text2MDT task while maintaining a lightweight architecture. |
PUMA: Projected Universal Multilingual ASR for Low-Resource Settings. Application to Diverse African Languages (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing multilingual ASR models fail to generalize to low-resource languages while remaining costly to scale. |
| Approach: | They propose a multilingual ASR model that integrates a learnable language token with acoustic representations to enable language-aware processing. |
| Outcome: | The proposed model improves low-resource performance with reduced model complexity on African languages. |
Universal Sentence Encoder for English (D18-2)
Copied to clipboard
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, Ray Kurzweil
| Challenge: | TensorFlow Hub sentence embedding models have good task transfer performance . model variants allow for trade-offs between accuracy and compute resources . |
| Approach: | They propose easy-to-use TensorFlow Hub sentence embedding models with good task transfer performance. |
| Outcome: | The proposed models outperform models without transfer learning and those that use only word-level transfer on a number of NLP tasks. |
OodGAN: Generative Adversarial Network for Out-of-Domain Data Generation (2021.naacl-industry)
Copied to clipboard
| Challenge: | Existing models for OOD detection work with text, but they do not work directly with the text. |
| Approach: | They propose to use a sequential generative adversarial network (SeqGAN) based model to generate OOD data for a given domain automatically. |
| Outcome: | The proposed model outperforms state-of-the-art in OOD detection metrics for ROSTD and OSQ datasets. |
Character-Level Models versus Morphology in Semantic Role Labeling (P18-1)
Copied to clipboard
| Challenge: | Character-level models are used for high-level semantic analysis tasks such as semantic role labeling. |
| Approach: | They train character-level models that use word, character and morphology level information . they analyze how performance of characters compare to words and a variety of morphological typologies . |
| Outcome: | The results shed light on important characteristics of character-level models and their semantic capability. |
Teacher Intervention: Improving Convergence of Quantization Aware Training for Ultra-Low Precision Transformers (2023.eacl-main)
Copied to clipboard
| Challenge: | Quantization-aware training (QAT) is a promising method to lower the implementation cost and energy consumption. |
| Approach: | They propose a method for fast converging QAT of pre-trained Transformers using a layer-wise signal propagation method with the intact signal from the teacher. |
| Outcome: | The proposed method achieves superior accuracy with significantly lower fine-tuning iterations on well-known Transformers of natural language processing as well as computer vision compared to the state-of-the-art methods. |
Multimodal fusion via cortical network inspired losses (2022.acl-long)
Copied to clipboard
| Challenge: | Recent work in deep fusion models has led to substantial improvements over unimodal approaches in areas like speech recognition, emotion recognition and analysis. |
| Approach: | They propose to introduce neural dependencies into the loss functions to allow for fusion of different modalities while keeping the model complexity manageable. |
| Outcome: | Experiments on multimodal sentiment analysis tasks show that the proposed approach provides a consistent performance boost. |
In-Context Analogical Reasoning with Pre-Trained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Analogical reasoning is a fundamental capacity of human cognition that allows us to reason abstractly about novel situations by relating them to past experiences. |
| Approach: | They apply large pre-trained language models to visual Raven’s Progressive Matrices (RPM) and use language-based abstractions to support analogy in AI systems. |
| Outcome: | The proposed language-based abstractions outperform human models on Raven’s Progressive Matrices and supervised vision-based methods. |
Sequicity: Simplifying Task-oriented Dialogue Systems with Single Sequence-to-Sequence Architectures (P18-1)
Copied to clipboard
| Challenge: | Existing solutions to task-oriented dialogue systems follow pipeline designs which introduces complexity and fragility. |
| Approach: | They propose a novel sequence-to-sequence (seq2sequ) model which tracks dialogue believes and a two stage copynet instantiation which emonstrates good scalability. |
| Outcome: | The proposed framework outperforms state-of-the-art pipeline-based methods on large datasets and retains satisfactory entity match rate on out-of vocabulary (OOV) cases where pipeline-designed competitors totally fail. |
Star-Transformer (N19-1)
Copied to clipboard
| Challenge: | Existing models with fully-connected attention connections are heavy and require large training data. |
| Approach: | They propose a lightweight alternative to the Transformer by sparsifying the fully-connected structure with a star-shaped topology. |
| Outcome: | The proposed model achieves significant performance improvements on 22 datasets on four tasks. |
Lightweight, Dynamic Graph Convolutional Networks for AMR-to-Text Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | AMR-to-text generation is used to transduce Abstract Meaning Representation structures (AMRs) Graph Convolution Networks (GCNs) are not able to capture non-local information and follow a local (first-order) information aggregation scheme. |
| Approach: | They propose a dynamic fusion mechanism that captures richer non-local interactions . they propose weight tied convolutions and group graph convolution to reduce memory usage . |
| Outcome: | The proposed model outperforms state-of-the-art models on two benchmark datasets with significantly fewer parameters while maintaining the model capacity. |
EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets (2021.acl-long)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models require enormous computational resources and long training time. |
| Approach: | They propose an algorithm to reduce inference time and train large NLP models by slimming the self-attention and fully-connected sub-layers inside a transformer. |
| Outcome: | The proposed algorithm achieves comparable performance to standard BERT with 35 45% less training time. |
UniBridge: A Unified Approach to Cross-Lingual Transfer Learning for Low-Resource Languages (2024.acl-long)
Copied to clipboard
| Challenge: | Existing pre-trained language models are weak in addressing cross-lingual transfer tasks. |
| Approach: | They propose a method for initializing embeddings and choosing the right vocabulary size for cross-lingual systems. |
| Outcome: | The proposed method improves the F1-Score in several languages . |
Retrieval-Augmented Controllable Review Generation (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to generate reviews using attribute identifiers are limited and dependent on how well they can capture vector representations of attributes. |
| Approach: | They propose to leverage attributes as inputs for review generation by using reference sets . they propose to use these references to enrich inductive biases of given attributes . |
| Outcome: | The proposed model improves over previous approaches on automatic and human evaluation metrics. |
Semi-Supervised Dependency Parsing with Arc-Factored Variational Autoencoding (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for dependency parsing use unlabelled data to compensate for the lack of training corpora. |
| Approach: | They propose semi-supervised dependency parsing methods that utilize unlabelled data to compensate for the scarcity of labelled training corpora. |
| Outcome: | The proposed model overcomes the tree constraint and the complexity of the training procedure while avoiding the challenges brought by the tree constraints. |
SEEK: Segmented Embedding of Knowledge Graphs (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for knowledge graph embedding can not make a proper trade-off between the model complexity and the model expressiveness, which makes them far from satisfactory. |
| Approach: | They propose a lightweight modeling framework that can achieve highly competitive relational expressiveness without increasing the model complexity. |
| Outcome: | The proposed framework can achieve highly competitive relational expressiveness without increasing model complexity. |
Comparative evaluation of boundary-relaxed annotation for Entity Linking performance (2023.acl-long)
Copied to clipboard
| Challenge: | Entity Linking is a critical step for information extraction, allowing the retrieval and understanding of information from unstructured textual sources. |
| Approach: | They propose to use noisy datasets to generate noisy versions of annotated entity mentions and then train three Entity Linking models on this data. |
| Outcome: | The proposed model can be used to associate NE mentions to a single concept in an ontology, allowing for better indexing and relation extraction. |
Tied-LoRA: Enhancing parameter efficiency of LoRA with Weight Tying (2024.naacl-long)
Copied to clipboard
| Challenge: | a new paradigm for low-rank Adaptation (LoRA) uses weight tying and selective training to improve parameter efficiency. |
| Approach: | They propose a paradigm that uses weight tying and selective training to enhance parameter efficiency of Low-rank Adaptation. |
| Outcome: | The proposed paradigm achieves comparable performance to LoRA with reduced model complexity . the proposed paradigm can be used for a variety of tasks and languages . |
HIT: Nested Named Entity Recognition via Head-Tail Pair and Token Interaction (2020.emnlp-main)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a fundamental task in natural language processing due to the nature of the named entity. |
| Approach: | They propose a nested NER model that leverages two key properties pertaining to the named entity, including explicit boundary tokens and tight internal connection between tokens within the boundary. |
| Outcome: | The proposed model achieves state-of-the-art on three public NER datasets. |
A Lightweight Method to Generate Unanswerable Questions in English (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to build robust question answering models are too complex . antonym and entity swaps on answerable questions are used to build models . |
| Approach: | They propose a method for performing antonym and entity swaps on unanswerable questions. |
| Outcome: | The proposed method outperforms the previous state-of-the-art and has higher human-judged relatedness and readability. |
Ensemble Transformer for Efficient and Accurate Ranking Tasks: an Application to Question Answering Systems (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Large transformer models are expensive and slow to use in many applications. |
| Approach: | They propose an efficient neural network to distill large transformers into a single smaller model. |
| Outcome: | The proposed model outperforms existing models on English datasets . it outperformed existing models with 2.7 more parameters and 2.5 slower . |
ESC: Efficient Speech Coding with Cross-Scale Residual Vector Quantized Transformers (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing neural speech codecs trade model complexity for reconstruction performance . ESC is a lightweight, parameter-efficient speech coder . |
| Approach: | They propose an efficient speech codec based on a cross-scale residual vector quantization scheme and transformers that can achieve high-fidelity speech reconstruction with significantly lower model complexity. |
| Outcome: | The proposed codec achieves high-fidelity speech reconstruction with significantly lower model complexity. |
Revisiting Over-Smoothness in Text to Speech (2022.acl-long)
Copied to clipboard
| Challenge: | Non-autoregressive text to speech models ignore correlation in time and frequency domains, causing blurry results. |
| Approach: | They revisit the problem of over-smoothness in non-autoregressive text to speech models . they use methods that reduce complexity of data distributions and improve modeling methods . |
| Outcome: | The proposed models achieve better voice quality and faster inference speed than autoregressive models. |
Two-Stage Fine-Tuning for Improved Bias and Variance for Large Pretrained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work challenges the bias-variance trade-off . large pretrained models can have large variance and overfit domain-specific data . |
| Approach: | They propose a bias-variance trade-off that implies learning methods need to balance complexity with data size to minimize under-fitting and over-fit. |
| Outcome: | The proposed method achieves strong results on SuperGLUE and clinical information extraction tasks. |
Negative Matters: Multi-Granularity Hard-Negative Synthesis and Anchor-Token-Aware Pooling for Enhanced Text Embeddings (2025.acl-long)
Copied to clipboard
| Challenge: | Text embedding models are used for various natural language processing tasks such as sentiment analysis, text clustering, and content-based information retrieval. |
| Approach: | They propose a synthesis framework that leverages large language models to generate diverse negative samples with varying levels of similarity with the query. |
| Outcome: | The proposed framework achieves state-of-the-art performance surpassing existing synthesis strategies with synthetic data and when combined with public retrieval datasets. |